1. Field of the Invention
The present invention relates to a method and apparatus for providing automated, coordinated accompaniment with respect to a performance and, more specifically, automated, coordinated accompaniment for a performance by an ensemble.
2. Description of the Invention Background
Several systems for following and accompanying solo performers have previously been developed and described in the computer music literature. If all performers were perfect, then coordinating and synchronizing an automated accompaniment with an ensemble would be no more difficult than coordinating and synchronizing an automated accompaniment with a polyphonic instrument such as a piano. In reality, ensemble players are not necessarily very well synchronized, and some players may become lost or consistently drag or rush the tempo. Even when a performance goes well, individual players will rest and rejoin the ensemble as indicated in the score. During contrapuntally active sections of a composition, some performers may play moving lines while others sustain tones. An ensemble player must integrate that information and resolve contradictions to form a sense of the true ensemble tempo and score position.
The problem of solo accompaniment can be partitioned into three distinct sub-problems:
1. reliably detecting what the soloist has performed, PA1 2. determining the score position of the soloist from the detected performance, and PA1 3. producing an accompaniment in synchrony with the detected performance.
Likewise, a system which attempts to responsively accompany an ensemble of live performers must also address each of those tasks. Furthermore, the system's capabilities must be extended beyond those of the solo accompaniment system so that it is able to track multiple performers simultaneously. That corresponds to the multiple, simultaneous execution of reliable performance detection and determination of score position. Before taking actions to control production of the accompaniment, the system must also integrate the information derived from tracking each performer. That may require resolution of discrepancies, such as different estimates of the individual performer's score positions. In addition to problems encountered in tracking multiple performers and resolving discrepancies, problems are also encountered in reliably detecting what has been performed by the ensemble.
In dealing with the first of the three subproblems, i.e. reliably detecting what has been performed, the goal is to extract from the performance important musical parameters that can be used to determine the score position and tempo of each member of the ensemble. Such parameters might include fundamental pitch, note duration, attack, dynamic (relative loudness), and articulation. The precise parameters obtained by a particular system could vary according to the type of performance it tracks, how that performance is represented, and the expense and reliability with which certain parameters may be extracted from the representation. In the simplest case, MIDI messages sent from an electronic keyboard can provide very reliable, clean, and precise information about the pitch, duration, and dynamic of the performance. That information can be inexpensively extracted from such a representation.
In a more difficult case, the representation of the performance might be an audio signal from a microphone. That is often the case when tracking the performance from acoustic wind instruments. A system which must extract parameters from a representation like this will need to devote more computational time to analysis of the signal and deal with issues like background noise, and distinguish between signals received from onset transients versus sustained pitches. A yet more difficult case is to distinguish multiple instruments recorded with a single microphone. For performances from different instruments, the system may need to track different parameters. Tracking vocal performances presents its own unique set of problems.
The second task of an accompaniment system is tracking the score position of performers in real-time. That involves matching sequences of detected performance parameters to a score. The "score" might exist in a variety of forms, including a completely composed piece or simply an expected harmonic progression. Several considerations complicate tracking score location of multiple performers. First, the tracking needs to be accomplished efficiently so that the system is able to control the accompaniment in real-time. The more quickly a system can recognize that a soloist has entered early, for example, the more quickly it will be able to adjust the accompaniment performance to accommodate. Additionally, since a flawless performance is not guaranteed, the tracking process must be tolerant of extraneous parameters as generated by an occasional wrong note, extra note, or omitted note. Finally, the method chosen to track the performer must use, in an appropriate manner, the parameters extracted by the performance analyzer. For example, if the performance analyzer can reliably recognize pitch signals and extract the fundamental, but is not so reliable at recognizing attacks, the tracking system should be appropriately less dependent upon the attack information.
If successive score locations can be accurately identified in the performance and time-stamped, then the accompaniment system may be able to derive accurate tempo predictions. However, it is well-known that performers alter durations of notes for expressive purposes. Accompaniment systems must recognize such artistic variations so as to avoid sudden drastic tempo changes while at the same time remaining capable of reacting to actual, expressive tempo changes initiated by the performers. Reacting too slowly or too hesitantly to such legitimate changes can noticeably detract from the overall performance.
Once estimates of individual performers' score locations and tempi are obtained, an ensemble accompaniment system must combine and resolve that information. Before the system can make adjustments to the accompaniment performance, attempting to synchronize with the ensemble, it must have an idea of the overall ensemble score position and tempo. Several considerations affect generation of those estimates such as the reliability with which a performance is tracked. That would be the case if a performer's input signal is noisy and difficult to analyze, or for performers who make numerous mistakes (wrong notes, omitted notes, etc.) such that their score position cannot reliably be determined. A second consideration is the activity of the performer. A performer sustaining a long note may not provide as much information about score position as does a performer whose output is continually changing. Finally, the accompaniment system must be able to handle performers who are presumably lost, possibly having fallen behind or made an entrance too early.
Having obtained estimates of ensemble score position and tempo which take into account those considerations as much as possible, an accompaniment system must than decide when and how to adjust the accompaniment performance. Generally, an accompaniment must be continuous and aesthetically acceptable, yet reactive to the performers' omissions, errors, and tempo changes. If the performers increase or decrease the tempo for interpretive reasons, the accompaniment system should do likewise. If the performers pause or jump ahead in the score, then the accompaniment should follow as much as possible, but should always sound "musical" rather than "mechanical". The system must react to the performers' actions in a generally expected and reasonable fashion.
Thus, the need exists for reliably detecting what has been performed by an ensemble during a live performance and for coordinating and synchronizing an automated accompaniment therewith.